A comprehensive and systematic model of user evaluation of Web search engines: II. An evaluation by undergraduates
Identifieur interne : 000768 ( Main/Exploration ); précédent : 000767; suivant : 000769A comprehensive and systematic model of user evaluation of Web search engines: II. An evaluation by undergraduates
Auteurs : Louise T. Su [République populaire de Chine]Source :
- Journal of the American Society for Information Science and Technology [ 1532-2882 ] ; 2003-11.
English descriptors
- Teeft :
- Academic orientation, Alta, Alta vista, American society, Anova, Anova results, Anova tests, Boolean, Borgman, Browser, Common usage, Complete relevance, Comprehensiveness, Computer experience, Computer ownership, Connectivity, Content analysis, Current study, Database, Discipline interaction, Engine discipline interaction, Entire sample, Good links, Good results, Graduate schools, Helpful results, Hotbot, Humanities, Humanities undergraduates, Information science, Infoseek, Infoseek lycos, Interface, Internet, Internet experience, Invalid links, Irrelevant hits, Keywords, Kruskal, Kruskal wallis tests, Lycos, Main effect, Measure engine, Nding, Ndings, Negative comments, Netscape, Online, Online documentation, Other engines, Other software, Output display, Overall performance, Participant, Participant experiences, Personal interests, Positive comments, Previous study, Qualitative data, Quantitative analysis, Quantitative data, Query, Relative performance, Relevance measures, Relevant documents, Relevant hits, Relevant information, Relevant items, Response time, Retrieval, Retrieving, Satisfaction ratings, Sciences undergraduates, Search comprehensiveness, Search engines, Search interface, Search options, Search queries, Search requirements, Search results, Search strategy, Search time, Searcher, Second method, Social sciences, Software, Spearman, Srivastava, Standard deviations, Subject expertise, System features, Time period, Total sample, Tukey, Tukey post, User, User criteria, User evaluation, User satisfaction, User satisfaction measures, Valid links, Verbal data, Vista, Wallis.
Abstract
This paper presents an application of the model described in Part I to the evaluation of Web search engines by undergraduates. The study observed how 36 undergraduate used four major search engines to find information for their own individual problems and how they evaluated these engines based on actual interaction with the search engines. User evaluation was based on 16 performance measures representing five evaluation criteria: relevance, efficiency, utility, user satisfaction, and connectivity. Non‐performance (user‐related) measures were also applied. Each participant searched his/her own topic on all four engines and provided satisfaction ratings for system features and interaction and reasons for satisfaction. Each also made relevance judgements of retrieved items in relation to his/her own information need and participated in post‐search interviews to provide reactions to the search results and overall performance. The study found significant differences in precision PR1, relative recall, user satisfaction with output display, time saving, value of search results, and overall performance among the four engines and also significant engine by discipline interactions on all these measures. In addition, the study found significant differences in user satisfaction with response time among four engines, and significant engine by discipline interaction in user satisfaction with search interface. None of the four search engines dominated in every aspect of the multidimensional evaluation. Content analysis of verbal data identified a number of user criteria and users evaluative comments based on these criteria. Results from both quantitative analysis and content analysis provide insight for system design and development, and useful feedback on strengths and weaknesses of search engines for system improvement.
Url:
DOI: 10.1002/asi.10334
Affiliations:
Links toward previous steps (curation, corpus...)
- to stream Istex, to step Corpus: 001952
- to stream Istex, to step Curation: 001952
- to stream Istex, to step Checkpoint: 000650
- to stream Main, to step Merge: 000770
- to stream Main, to step Curation: 000768
Le document en format XML
<record><TEI wicri:istexFullTextTei="biblStruct"><teiHeader><fileDesc><titleStmt><title xml:lang="en">A comprehensive and systematic model of user evaluation of Web search engines: II. An evaluation by undergraduates</title>
<author><name sortKey="Su, Louise T" sort="Su, Louise T" uniqKey="Su L" first="Louise T." last="Su">Louise T. Su</name>
</author>
</titleStmt>
<publicationStmt><idno type="wicri:source">ISTEX</idno>
<idno type="RBID">ISTEX:F3C5ABC3A6DB016C093BFDEF05C6804F6E718BEB</idno>
<date when="2003" year="2003">2003</date>
<idno type="doi">10.1002/asi.10334</idno>
<idno type="url">https://api.istex.fr/ark:/67375/WNG-QNZ931P3-W/fulltext.pdf</idno>
<idno type="wicri:Area/Istex/Corpus">001952</idno>
<idno type="wicri:explorRef" wicri:stream="Istex" wicri:step="Corpus" wicri:corpus="ISTEX">001952</idno>
<idno type="wicri:Area/Istex/Curation">001952</idno>
<idno type="wicri:Area/Istex/Checkpoint">000650</idno>
<idno type="wicri:explorRef" wicri:stream="Istex" wicri:step="Checkpoint">000650</idno>
<idno type="wicri:doubleKey">1532-2882:2003:Su L:a:comprehensive:and</idno>
<idno type="wicri:Area/Main/Merge">000770</idno>
<idno type="wicri:Area/Main/Curation">000768</idno>
<idno type="wicri:Area/Main/Exploration">000768</idno>
</publicationStmt>
<sourceDesc><biblStruct><analytic><title level="a" type="main" xml:lang="en">A comprehensive and systematic model of user evaluation of Web search engines: II. An evaluation by undergraduates<ref type="note" target="#fn1"></ref>
</title>
<author><name sortKey="Su, Louise T" sort="Su, Louise T" uniqKey="Su L" first="Louise T." last="Su">Louise T. Su</name>
<affiliation wicri:level="4"><country xml:lang="fr" wicri:curation="lc">République populaire de Chine</country>
<wicri:regionArea>Formerly Assistant Professor, University of Pittsburgh, Pittsburgh, PA 15260; 593 Wenhwa Road, Rende Shiang, Tainan, Taiwan 717</wicri:regionArea>
<orgName type="university">Université de Pittsburgh</orgName>
<placeName><settlement type="city">Pittsburgh</settlement>
<region type="state">Pennsylvanie</region>
</placeName>
</affiliation>
<affiliation></affiliation>
</author>
</analytic>
<monogr></monogr>
<series><title level="j" type="main">Journal of the American Society for Information Science and Technology</title>
<title level="j" type="alt">JOURNAL OF THE AMERICAN SOCIETY FOR INFORMATION SCIENCE AND TECHNOLOGY</title>
<idno type="ISSN">1532-2882</idno>
<idno type="eISSN">1532-2890</idno>
<imprint><biblScope unit="vol">54</biblScope>
<biblScope unit="issue">13</biblScope>
<biblScope unit="page" from="1193">1193</biblScope>
<biblScope unit="page" to="1223">1223</biblScope>
<biblScope unit="page-count">31</biblScope>
<publisher>Wiley Subscription Services, Inc., A Wiley Company</publisher>
<pubPlace>Hoboken</pubPlace>
<date type="published" when="2003-11">2003-11</date>
</imprint>
<idno type="ISSN">1532-2882</idno>
</series>
</biblStruct>
</sourceDesc>
<seriesStmt><idno type="ISSN">1532-2882</idno>
</seriesStmt>
</fileDesc>
<profileDesc><textClass><keywords scheme="Teeft" xml:lang="en"><term>Academic orientation</term>
<term>Alta</term>
<term>Alta vista</term>
<term>American society</term>
<term>Anova</term>
<term>Anova results</term>
<term>Anova tests</term>
<term>Boolean</term>
<term>Borgman</term>
<term>Browser</term>
<term>Common usage</term>
<term>Complete relevance</term>
<term>Comprehensiveness</term>
<term>Computer experience</term>
<term>Computer ownership</term>
<term>Connectivity</term>
<term>Content analysis</term>
<term>Current study</term>
<term>Database</term>
<term>Discipline interaction</term>
<term>Engine discipline interaction</term>
<term>Entire sample</term>
<term>Good links</term>
<term>Good results</term>
<term>Graduate schools</term>
<term>Helpful results</term>
<term>Hotbot</term>
<term>Humanities</term>
<term>Humanities undergraduates</term>
<term>Information science</term>
<term>Infoseek</term>
<term>Infoseek lycos</term>
<term>Interface</term>
<term>Internet</term>
<term>Internet experience</term>
<term>Invalid links</term>
<term>Irrelevant hits</term>
<term>Keywords</term>
<term>Kruskal</term>
<term>Kruskal wallis tests</term>
<term>Lycos</term>
<term>Main effect</term>
<term>Measure engine</term>
<term>Nding</term>
<term>Ndings</term>
<term>Negative comments</term>
<term>Netscape</term>
<term>Online</term>
<term>Online documentation</term>
<term>Other engines</term>
<term>Other software</term>
<term>Output display</term>
<term>Overall performance</term>
<term>Participant</term>
<term>Participant experiences</term>
<term>Personal interests</term>
<term>Positive comments</term>
<term>Previous study</term>
<term>Qualitative data</term>
<term>Quantitative analysis</term>
<term>Quantitative data</term>
<term>Query</term>
<term>Relative performance</term>
<term>Relevance measures</term>
<term>Relevant documents</term>
<term>Relevant hits</term>
<term>Relevant information</term>
<term>Relevant items</term>
<term>Response time</term>
<term>Retrieval</term>
<term>Retrieving</term>
<term>Satisfaction ratings</term>
<term>Sciences undergraduates</term>
<term>Search comprehensiveness</term>
<term>Search engines</term>
<term>Search interface</term>
<term>Search options</term>
<term>Search queries</term>
<term>Search requirements</term>
<term>Search results</term>
<term>Search strategy</term>
<term>Search time</term>
<term>Searcher</term>
<term>Second method</term>
<term>Social sciences</term>
<term>Software</term>
<term>Spearman</term>
<term>Srivastava</term>
<term>Standard deviations</term>
<term>Subject expertise</term>
<term>System features</term>
<term>Time period</term>
<term>Total sample</term>
<term>Tukey</term>
<term>Tukey post</term>
<term>User</term>
<term>User criteria</term>
<term>User evaluation</term>
<term>User satisfaction</term>
<term>User satisfaction measures</term>
<term>Valid links</term>
<term>Verbal data</term>
<term>Vista</term>
<term>Wallis</term>
</keywords>
</textClass>
</profileDesc>
</teiHeader>
<front><div type="abstract" xml:lang="en">This paper presents an application of the model described in Part I to the evaluation of Web search engines by undergraduates. The study observed how 36 undergraduate used four major search engines to find information for their own individual problems and how they evaluated these engines based on actual interaction with the search engines. User evaluation was based on 16 performance measures representing five evaluation criteria: relevance, efficiency, utility, user satisfaction, and connectivity. Non‐performance (user‐related) measures were also applied. Each participant searched his/her own topic on all four engines and provided satisfaction ratings for system features and interaction and reasons for satisfaction. Each also made relevance judgements of retrieved items in relation to his/her own information need and participated in post‐search interviews to provide reactions to the search results and overall performance. The study found significant differences in precision PR1, relative recall, user satisfaction with output display, time saving, value of search results, and overall performance among the four engines and also significant engine by discipline interactions on all these measures. In addition, the study found significant differences in user satisfaction with response time among four engines, and significant engine by discipline interaction in user satisfaction with search interface. None of the four search engines dominated in every aspect of the multidimensional evaluation. Content analysis of verbal data identified a number of user criteria and users evaluative comments based on these criteria. Results from both quantitative analysis and content analysis provide insight for system design and development, and useful feedback on strengths and weaknesses of search engines for system improvement.</div>
</front>
</TEI>
<affiliations><list><country><li>République populaire de Chine</li>
</country>
<region><li>Pennsylvanie</li>
</region>
<settlement><li>Pittsburgh</li>
</settlement>
<orgName><li>Université de Pittsburgh</li>
</orgName>
</list>
<tree><country name="République populaire de Chine"><region name="Pennsylvanie"><name sortKey="Su, Louise T" sort="Su, Louise T" uniqKey="Su L" first="Louise T." last="Su">Louise T. Su</name>
</region>
</country>
</tree>
</affiliations>
</record>
Pour manipuler ce document sous Unix (Dilib)
EXPLOR_STEP=$WICRI_ROOT/Wicri/Musique/explor/BourgeoisGentilV1/Data/Main/Exploration
HfdSelect -h $EXPLOR_STEP/biblio.hfd -nk 000768 | SxmlIndent | more
Ou
HfdSelect -h $EXPLOR_AREA/Data/Main/Exploration/biblio.hfd -nk 000768 | SxmlIndent | more
Pour mettre un lien sur cette page dans le réseau Wicri
{{Explor lien |wiki= Wicri/Musique |area= BourgeoisGentilV1 |flux= Main |étape= Exploration |type= RBID |clé= ISTEX:F3C5ABC3A6DB016C093BFDEF05C6804F6E718BEB |texte= A comprehensive and systematic model of user evaluation of Web search engines: II. An evaluation by undergraduates }}
This area was generated with Dilib version V0.6.33. |